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Preamble from the 
Institute of Education 
Sciences 

What is a practice guide? 

The health care professions have embraced a mecha- 
nism for assembling and communicating evidence- 
based advice to practitioners about care for specific 
clinical conditions. Variously called practice guidelines, 
treatment protocols, critical pathways, best-practice 
guides, or simply practice guides, these documents are 
systematically developed recommendations about the 
course of care for frequently encountered problems, 
ranging from physical conditions such as foot ulcers to 
socioemotional issues such as navigating the demands 
of adolescence . 1 

Practice guides are similar to the products of typical 
expert consensus panels in that they reflect the views 
of those serving on the panel, as well as the social 
decision processes that come into play as the positions 
of individual panel members are forged into statements 
that all are willing to endorse. However, practice guides 
are generated under three constraints that do not 
typically apply to consensus panels. The first is that a 
practice guide consists of a list of discrete recommenda- 
tions that are intended to be actionable. The second is 
that those recommendations taken together are intend- 
ed to comprise a coherent approach to a multifaceted 
problem. The third, which is most important, is that 
each recommendation is explicitly connected to the 
level of evidence supporting it (e.g., strong, moderate, 
and low). The levels of evidence are usually constructed 
around the value of particular types of studies for 
drawing causal conclusions about what works. Thus 
one typically finds that the top level of evidence is 
drawn from a body of randomized controlled trials, 
the middle level from well-designed studies that do not 
involve randomization, and the bottom level from the 
opinions of respected authorities. Levels of evidence 
can also be constructed around the value of particular 
types of studies for other goals, such as the reliability 
and validity of assessments. 



Practice guides can also be distinguished from 
systematic reviews or meta- analyses, which employ 
statistical methods to summarize the results of studies 
obtained from a rule-based search of the literature. 
Authors of practice guides seldom conduct the types of 
systematic literature searches that are the backbone of a 
meta- analysis, though they take advantage of such work 
when it is already published. Instead they use their 
expertise to identify the most important research with 
respect to their recommendations, augmented by a 
search of recent publications to assure that the research 
citations are up-to-date. Further, the characterization 
of the quality and direction of evidence underlying 
a recommendation in a practice guide relies less on a 
tight set of rules and statistical algorithms and more on 
the judgment of the authors than would be the case in 
a high-quality meta-analysis. Another distinction is that 
a practice guide, because it aims for a comprehensive 
and coherent approach, operates with more numerous 
and more contextualized statements of what works than 
does a typical meta-analysis. 

Thus practice guides sit somewhere between consensus 
reports and meta-analyses in the degree to which 
systematic processes are used for locating relevant 
research and characterizing its meaning. Practice guides 
are more like consensus panel reports than meta- 
analyses in the breadth and complexity of the topic 
that is addressed. Practice guides are different from 
both consensus reports and meta- analyses in providing 
advice at the level of specific action steps along a 
pathway that represents a more-or-less coherent and 
comprehensive approach to a multifaceted problem. 

Practice guides in education at the Institute 
of Education Sciences 

The Institute of Education Sciences (IES) publishes 
practice guides in education to bring the best avail- 
able evidence and expertise to bear on the types of 
systemic challenges that cannot be addressed by single 
interventions or approaches. Although IES has taken 
advantage of the history of practice guides in health 
care to provide models of how to proceed in education, 
education is different from health care in ways that 
may require that practice guides in education have 
somewhat different designs. Even within health care, 
where practice guides now number in the thousands, 



1 Field and Lohr (1990). 
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there is no single template in use. Rather, one finds 
descriptions of general design features that permit 
substantial variation in the realization of practice 
guides across subspecialties and panels of experts. 2 
Accordingly, the templates for IES practice guides 
may vary across practice guides and change over 
time and with experience. 

The steps involved in producing an IES-sponsored 
practice guide are first to select a topic, which is 
informed by formal surveys of practitioners and 
spontaneous requests from the field. Next, a panel chair 
is recruited who has a national reputation and up-to- 
date expertise in the topic. Third, the chair, working in 
collaboration with IES, selects a small number of panel- 
ists to co-author the practice guide. These are people 
the chair believes can work well together and have the 
requisite expertise to be a convincing source of recom- 
mendations. IES recommends that at least one of the 
panelists be a practitioner with considerable experience 
relevant to the topic being addressed. The chair and the 
panelists are provided a general template for a practice 
guide along the lines of the information provided in 
this preamble. They are also provided with examples of 
practice guides. The practice guide panel works under 
a short deadline of 6 to 9 months to produce a draft 
document. The expert panel interacts with and receives 
feedback from staff at IES during the development of 
the practice guide, but the panel members understand 
that they are the authors and thus responsible for the 
final product. 

One unique feature of IES-sponsored practice guides is 
that they are subjected to rigorous external peer review 
through the same office that is responsible for indepen- 
dent review of other IES publications. A critical task of 
the peer reviewers of a practice guide is to determine 
whether the evidence cited in support of particular 
recommendations is up-to-date, and that studies 
of similar or better quality that point in a different 
direction have not been ignored. Peer reviewers also are 
asked to evaluate whether the evidence grades assigned 
to particular recommendations by the practice guide 
authors are appropriate. A practice guide is revised as 
necessary to meet the concerns of external peer reviews 
and to gain the approval of the standards and review 
staff at IES. The process of external peer review is 



carried out independently of the office and staff within 
IES that initiated the practice guide. 

Because practice guides depend on the expertise of their 
authors and their group decision-making, the content 
of a practice guide is not and should not be viewed as 
a set of recommendations that in every case depends 
on and flows inevitably from scientific research. It 
is not only possible but also likely that two teams of 
recognized experts working independently to produce a 
practice guide on the same topic would generate 
products that differ in important respects. Thus con- 
sumers of practice guides need to understand that they 
are, in effect, getting the advice of consultants. These 
consultants should, on average, provide substantially 
better advice than educators might obtain on their 
own because the authors are national authorities who 
have to achieve consensus among themselves, justify 
their recommendations in terms of supporting 
evidence, and undergo rigorous independent peer 
review of their product. 
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2 American Psychological Association (2002). 
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Introduction 

The goal of this practice guide is to formulate specific 
and coherent evidence-based recommendations that 
educators can use to encourage girls in the fields of 
math and science. The target audience is teachers 
and other school personnel with direct contact with 
students, such as coaches, counselors, and principals. 
The practice guide includes specific recommendations 
for educators and the quality of evidence that supports 
these recommendations. 

We, the authors, are a small group with expertise 
on this topic. The range of evidence we considered 
in developing this document is vast, ranging from 
experiments, to trends in the National Assessment of 
Educational Progress (NAEP) data, to correlational and 
longitudinal studies. For questions about what works 
best, high-quality experimental and quasi-experimental 
studies, such as those meeting the criteria of the What 
Works Clearinghouse, have a privileged position. In all 
cases, we pay particular attention to findings that are 
replicated across studies. 

Although we draw on evidence about the effectiveness 
of specific practices, we use this information to make 
broader points about improving practice. In this 
document, we have tried to take findings from research 
or practices recommended by experts and describe 
how the use of this recommendation might actually 
unfold in school settings. In other words, we aim to 
provide sufficient detail so that educators will have 
a clear sense of the steps necessary to make use of the 
recommendation. 

A unique feature of practice guides is the explicit and 
clear delineation of the quality and quantity of evidence 
that supports each claim. To this end, we adapted a 
semi-structured hierarchy suggested by the Institute 
of Education Sciences. This classification system helps 
determine whether the quality and quantity of available 
evidence in support of a practice is of strong, moderate, 
or low quality. This system appears in table 1 below. 

Strong refers to consistent and generalizable evidence 
that an approach or practice causes improved 
performance in math or science among girls or that an 
assessment is reliable and valid. Moderate refers either 
to evidence from (a) studies that allow strong causal 
conclusions but which cannot be generalized with 



assurance to the target population because, for example, 
the findings have not been sufficiently replicated, or 
(b) studies that are generalizable but have more causal 
ambiguity than offered by experimental designs, for 
example, statistical models of correlational data or 
group comparison designs where equivalence of the 
groups at pretest is uncertain. For assessments, moder- 
ate refers to high quality studies from a small number 
of samples that are not representative of the whole 
population. Low refers to expert opinion based on 
reasonable extrapolations from research and theory on 
other topics and/or evidence from studies that do not 
meet the standards for moderate or strong evidence. 

For each recommendation, we include an appendix 
that provides more technical information about the 
studies and our decisions regarding level of evidence for 
the recommendation. To illustrate the types of studies 
reviewed, we describe one study in considerable detail 
for each recommendation. Our goal in doing this is to 
provide interested readers with more detail about the 
research designs, the intervention components, and 
how impact was measured. By including a particular 
study, we do not mean to suggest that it is the best 
study reviewed for the recommendation or necessarily 
an exemplary study in any way. 
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